Field of the Invention
The present invention relates to a restoration filter generation device and method, an image processing device, an imaging device, a restoration filter generation program, and a non-transitory computer-readable medium, and particularly relates to a technology of generating a restoration filter for performing a restoration process on an image captured via an optical system on the basis of a point spread function (PSF) or optical transfer function (OTF) of the optical system.
Description of the Related Art
The restoration process on the image captured is a process in which characteristics of degradation (PSF/OTF) caused by aberration and the like of the optical system (imaging lens) are founded in advance and a captured image (image having degradation) is subjected to a restoration process by use of a restoration filter generated on the basis of the PSF/OTF so as to restore to a high resolution image.
The PSF and the OTF are in relation of the Fourier transform, and the PSF is a function of a real variable and the OTF is a function of a complex variable. Examples of those having information equivalent to these include a modulation transfer function or amplitude transfer function (MTF) and a phase transfer function (PTF), each of which represents an amplitude component and phase component of the OTF. Combination of the MTF and the PTF has an amount of the information equivalent to the OTF or the PSF.
The restoration process using the restoration filter generated on the basis of the PSF/OTF may be classified roughly into a “frequency restoration” and a “phase restoration”. The “frequency restoration” is for equalizing MTF characteristics degraded by the optical system, and the “phase restoration” is for moving an image with frequency dependence such that an asymmetry PSF shape is restored to points as much as possible.
Since the aberration of the optical system varies depending on wavelength, it is ideally desirable to use a restoration filter different for each of color images of red (R), green (G) and blue (B). However, because of a large arithmetic load in the processes for the respective colors, it has been considered that only a luminance component having a large visual effect is to be subjected the restoration process.
Since the PSF shape with respect to the luminance cannot be defined, some criteria need to be considered in order to find a tradeoff such that a restoration performance is not adversely affected.
As a solution to this problem, Japanese Patent Application Laid-Open No. 2010-140442 (hereinafter referred to as PTL 1) discloses a method for finding the PSF with respect to a luminance component image in which a transformation formula for a luminance is used as it is to mix the PSFs for the respective colors of RGB.
In a case of performing the restoration process on the luminance, correction corresponding to magnification chromatic aberration correction cannot be performed, and thus the magnification chromatic aberration correction needs to be separately performed.
Japanese Patent Application Laid-Open No. 2012-129932 (hereinafter referred to as PTL 2) describes a technology in which the restoration process and the magnification chromatic aberration correction are separately performed.
In the image processing method described in PTL 2, a restoration filter is generated on the basis of a point spread function of which a difference between color components of the point spread function is reduced by way of a relative parallel translation, or a restoration filter is generated from an optical transfer function whose linear component corresponding to the magnification chromatic aberration is removed. In other words, a restoration filter is generated which performs only the “frequency restoration”, but does not perform the “phase restoration”. Then, an input image is subjected to the restoration process by use of the generated restoration filter, and thereafter, a color shift corresponding to the magnification chromatic aberration of the image after the restoration process is detected and a correction (magnification chromatic aberration correction) is performed to reduce the detected color shift.
As another embodiment, the magnification chromatic aberration correction is performed on an input image by use of a color shift correction amount depending on the magnification chromatic aberration of design of the optical system, and subsequently, the restoration process is performed by use of the restoration filter performing only the “frequency restoration”. However, since the color shift possibly remains due to an image degrading factor such as variations in manufacturing or variations in light source spectra of the optical system even if the restoration process is performed by use of the above magnification chromatic aberration correction and restoration filter, the color shift of the image after the restoration process is detected and the magnification chromatic aberration correction is performed to reduce the detected color shift.